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ABSTRACT 



the administrative series of 
vocational education 
intended for use in classes 



or 
by 



One of five modules in 
thte 16-module series designed to train 
curriculum specialists, this module is 

individual study arrangements at the preservice or inservice level 
students with -varying amounts of experience in vocational educati* 
(These modules are revised versions of earlier study guides — see 
note.) Introductory, materials include an overview, instructions to 
the learner, detailed list of behavioral goals and objectives, and 
resources needed to «rf6mplete learning activities. The module is 
divided i.nto three sections , each based on one of the goals. Section 
1 defines tfie role of the curriculum specialist in evaluating 
vocational education curricula. Evaluation is defined, three 
approaches to evaluation are described, and three areas in vocational 
education curriculum evaluation are discussed. The other two' sections 
describe the purposes and major activities involved i* evaluating a 
vocational education curriculum prior to its implementation (section 
2) and after it has been implemented (section 3). Each section 
follows a standard format: text, individual study activities, 
discussion questions, and\group activities. A summary of the module 
follows. Appendixes include suggested responses to the study 
activities, a s^lf-check, responses to the self-check, and 
recommended references. (TLB) 
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Introduction 



* Evaluation is an integral part of the curriculum 
development process. In fact, it provides a vital monitor- 
ing' system that begins in* the curriculum planning stage, 
provides feedback during the development and early imple- 
mentation stages, and provides essential information to 
decision-makers regarding^i^ie ultimate outcomes of a cur- 
riculum. Evaluation shoulxl be an ongoing process that 
provides data for management decisions both during and 
after implementation* Its intent is to prove outcomes, to 
improve the curriculum prior to and "during its early 
implementation, and to ensure that new curricula are of 
the highest caliber. / . 

The relatively recent call for accountability in edu- 
cation has required that teachers, students, administra- 
tors, institutions, and tjie curriculum itself be evaluated. 
The vocational education curriculum specialist must be 
familiar with' various "types of evaluation, be^ able to con- 
duct or supervise curriculum evaluations, and know how to 
use evaluation data to improve the curriculum.* 

r 

This is one of two modules pertaining to the evalua- « 
tion of vocational education curricula. This module pro- 
vides an overview of evaluation concepts and cjiscus^es- the 
principles and methods used to evaluate the< short-t£nn out- 
comes of vocational education curricula. The other evalua- 
tion module deals with conducting follow-up studies to 
evaluate the longer-term outcomes of vocational education 
curricula atid with communicating and usinfe evaluation 
results. 



Overview m * 

The first goal of ^:he module is concerned with the 
role of the curriculum Specialist in evaluating vocational 
education curricula. An attempt is made to define "evalua: 
tion" and to describe three general approaches t^> evalua- 
tion that have been applied to vocational education. The 
Comprehejasive Frogram Evaluation Approach is introduced as 
the framework for this jnpdule's approach ,to vocaj^orial 
education curriculum evaluation. 

The module emphasized a few specific areas involved 
in curriculum evaluation, since it is beyond its scope to 
provide through preparation in- comprehensive program 
evaluation skills. The importance of assessing the plan- 
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ning and development of the curriculum, the way it is 
being used, and the outcomes it produces are briefly 
discussed. 

The .remainder of the module concentrates on evaluat- 
ing the short-term outcomes of vocational education curri- 
cula. Two phases of evaluation are discussed in Goals 2 
and 3 respectively: the pre-implementation and implemen- 
tation phases.' C — » 

Prior to perfojrming any evaluation, several analyses 
should be performed to /determine the worth and measur- 
ability of the intended, curriculum outcomes and to de'ter- 
mine the relationship between the intended outcomes and 
the proposed activities of the curriculum. When these, 
analyses produce satisfactory, (af f irmative results, atten- 
tion must be directed to pre-implementation evaluation 
activities or tryouts. 

The' importance of and need for tryout activities is 
emphasized in the module, and a general strategy for their 
conduct is outlined. Guidelines fo,r developing measures 
of process and of intended and unintended outcomes are 
presented along wi^fch suggested evaluation designs and sam- 
pling strategies. Simple data processing and analysis 
techniques are mentioned. 

The third goal provides greater detail on evaluation 
methodology in the context of evaluating vocational educa- 
tion curricula after they have been implemented. Likely, 
questions to be answered by such an evaluation are posed. 
Details are presented on planning the evaluation, develop-* 
ing measures, selecting a design and sample, conducting 
the evaluation, and processing and analyzing the data. 
Special attention is paid to five criteria of adequacy 
that tnust.be considered when deye'lop.ing measures.. 



tp the Learner 

The Self-Check items and possible responses to th'em 
are foutid in the appendices. These questions "have tv^o 
purposes. First, before you begin work on the module, you 
may use them fcq^check quickly whether you v have already 
-learned the information in previous classes or readings. 
In some instances, with the consent of your instructor, 
you might decide t<^iskip a whole module or parts of "one. 
The Second purpose of the Self-Check is to help you review 
the CQntent of module^ you have studied in order to assess 



-10- 4 ' 

10 



whether you have achieved the module's goals and objec- 
tives. ^ 

You can also use the list of goals and objectives 
that follows to determine whether the module content is 
new to you and requires in-depth, study, or whether -the 
module can sewe as a brief review before you continue to 
the next module. 



s 
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Go4ls and Objectives ' 

7 * 

i * 

Go^l 1 : Define tjie role of the curriculum specialist in evalu- 
ating vocational education. f 

) 

\ Objective 1>1 : Summarize three apprpaches to vocational 
' education evaluation. \ 4 

>»* 

Objective 1.2 : Delineate the curriculum specialist^ role 
in vocational education evaluation. 

** i . 

Objective 1.3 : Describe three essential areas xn the 

evaluation, of a vocational education curriculum. 

Gogl -2 : Describe the'purposes and major activities involved in 
evaluating a vocational education curriculum prior to its 
implementation. » 

* ♦ 

" Objective 2.1 : Describe thrJe activities that should be 
performed 'as preliminary steps in evaluating a vocational 
•education curriculum. 

Objective 2.2 : State the main purpose of conducting pre- 
liminary tryouts '(pilot tests) of a vocational education 
curriculum and list the five step's involved. • 

Objective 2.3 : Identify two basic measurement approaches 
likely to be used in an evaluation anfl at least two types 
of^ tests used in each approach. 
+ 

Objective 2,t<fc Describe the evaluation design and sample 
size, most appropriate fdr use-i^a pre-implementation 
evaluation. 

Objective 2.5 : List appropriate methods for processing , 
and analyzing evaluation data and possible uses for th^e 
data. 



Goal 3 : Describe the purposes and major activities involved in 
evaluating^ vocational education curriculum after it has been 

implemented. ' 

♦ y * 

Objective 3.1 : State the purposes of conducting- an 
evaluation after the curriculum has been implemented and 
list the five steps involved. 

\ 

Objective 3.2 : Identify the characteristics of good mea- 
surement techniques. 
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; Objecti^e 3.3 / Describe the evaluation design and^sample 
size. most appropriate for use irt an implementation 
evaluation. . m 

Objective 3*4 : List appropriate methods for processing 
and analyzing* evaluation data* 

4 



Resources 

) ' ■ 

In order t?o complete the learning activities in this 
module, you will need information contained in the following 
publications : 

Finch, C, R. , & Crunkilton, J, R. Curriculum development 
in vocational and jfeechnipal education: Planning, con - 
tent, and implementation . Boston: Allyn and Bacon, 
1979. 

Wentling, T. L. Evaluating occupational education and 
training programs . Boston: Allyn and Bacon, 1980. 
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GOJ^U \i Define the r<51e of the cifj'ficulum specialist in evalu- 
„ ~ ^ ating vocational education. i 



Defining Evaluation 



Abramson (1979), in a review of the literature concerned 
with the definition of evaluation, concluded that M a consensus 
definition 11 for tb-ifc field of endeavor does not exist. The 
various types of definitions he identified were: (1) -measure- 
ment; (2) congruence between performance and objectives; 
(3) professional judgment; (4) description ; and (5) provision 
of information for decisionmaking. Evaluation was also distin- 
guished from evaluative research, which is a more "scientific 11 



xs a more 
.uing. \ 



process aimed at proving rather than valu 

• » ** 

The "valuing 11 concept provides a clue to the difference - 
between two major current definitions of evaluation discussed 
by Wentling (1979), the first being the provision of information 
for judging decision alternatives, and the second being the 
determination of worth. According to Abramson, the first defi- 
nition stresses collecting data to present to decision makers, 
and the second requires that value judgment company the data. 

* m * 4 \ 

Because evaluation means different things to different 
people, it is necessary to define the type of evaluation that 
will be the focus of this module. The initial step in this - 
defining process is to describe three evaluation approaches 
that have been applied to * vocational education. 



Evaluating Vocational Education 

The purposes and procedures of evaluation activities that 
have been applied to vocational education can be' grouped into 
three general approaches that will be discussed below. While a 
particular evaluation probably cannot be neatly categorized as 
one approach or another, the following, classification scheme 
can serve to characterize it or, at least, some of its elements. 



Accreditation approach . The accreditation concept has been 
around since the 1800s, when it was introduced as a means of 
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regulating the quality of education offered in the professions, 
mainly medicine. Today many organizations, including regional ( 
and professional associations, state agencies, and single pur- 
pose groups, conduct activities called accreditation. An entire 
educational institution or a single educational program may ^e 
accredited. Typically, the process is described as being 
voluntary, but it may, in fact, be mandatory. Usually accredi- 
tation must be renewed periodically; often this is every'five 
to ten years. 

The accreditation process is ^usually characterized by two 
phases, the sel'f-study and the site visit. During the self- 
study phase an institution or program examines its offerings in 
light of genera? criteria, identifying strengths and weaknesses. 
A team. of experts then examines the self-study report and yisits 
the institution or program to form opinions about its quality. 
If this peer review finds that the institution or program -meets 
minimum standards and Appears to be t meeting its own goal's, then 
accreditation is granted. 

The accreditatioh approach to evaluation will not be dis- 
cussed further, although many of the techniques fcrtr dat'a col-* 
lection, analysis, and reporting presented later in this module 
and in the following or^> are appropriate for self-studies con- 
ducted by vocational^education institutions or programs. 



Accountability approach . Funding agencies at all levels 
are demanding evidence that money used for vocational education 
is wisely spent for intended purposes. The defining feature of 
the accountability approach 'to evaluation is its attempt to 
provide thts information, usually in the form of aggregated 
data on students served, jobs attained, starting salaries, cost 
of instruction, *nd the like. ^ ^ 

Requirements associated with federal funds for vocational 
education have stimulated activity in the data collecting and 
reporting aspects of apcountability ^evaluation. Federal data 
needs stem from three aspects of the federal role in vocational 
•education evaluation: Establishing a need; ensuring funds are 
spent as intended, and urging program^ improvement and redirec- 
tion (Datta, 1979). The Vocational Education Act of 1963 and 
its 1968 amendments prompted many states to improve reporting 
practices, .which. resulted in a ripple effect that posed addi- 
tional requirements at the local level. However, this activity 
was not sufficient to overcome problems with national vocational 
education data stemming mainly from a lack of uniform defini- 
tions (Grasso; 1 1979, U.S. Department of Labor, 1978). 
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The federal -s tpte vocational education d.ata system (VEDS) 
called for in Title II (Vocational Education) of the Education 
Amendments of 1976 is an attempt to obtain reliable, accurate, 
and comparable data on vocational education students, programs, 
graduates and dropouts, staff, facilities, and expenditures 
(NCES$f 19.79)., Implementation of a system such as this calls* 
for ^^ew round of state and local reporting requirements. 

Accountability reporting forms are accompanied by instruc- 
tions; both forms and iftstructions vary from state to state and 
change* from year to year. When faced with unclear account- 
ability requirements, reasonable advice would be to seek tech- 
nical assistance. " - » 



CoiftprehensiVe program evaluation approach . The literature 
cm educational evaluation contains numerous models that can 
help structure the evaluation of atftotal vocational J&ducation 
program. AbramSQn (1979) contains a nicely organized review of 
them organized into the categories of decision-oriented models, 
judgment-oriented models,* and other models. 

The CIPP model* (Context, 1 I^nput, Process, Product) is 
prominent in the educational evaluation literature because of 
its Weadth and decision orientation. In theory, the model 
structures evaluation decisions along two dimensions (ends vs. 
means and intentions vs. actualities) which results in four 
types of evaluations Thes* are: 

• context evaluation to assist planning decisions about 
intended ends ; 

* • * input evaluatiqp to assist structuring decisions about 
intended ifteatas; 



process evaluation to* assist implementing dec?isiorfs 
kbout actual means; and * 

• product evaluation to assist recycling decisions about 
actual ends (Abramson, 1979)., V 

Wentling (1979S>Vlias translated some of the concepts of 
process and product evaluation into a system for program 

Revaluation, which, with minor modifications, can also be fc* 
apglied to the evaluation of courses or segments of instruc- 
tion. * Wentling f .s evaluation model focuses on eight areas of 

. concern: ' - > 



• administrative or management organization; r 
4 personnel; 

• objectives; 

• evaluation -system; / 

• content; - * . 

• learners "being served; 

• utilization of) resources; and 

• guidance, personnel counseling, placement, and other 
ancillary services of the program. 

Staff or personnel evaluati^ is an activity that has 
received minimal attention in education, although^it ^wide- 
spread in/those businesses and industries concerned with 
ijgfceased efficiency, production, and profit. Staff evaluation 
in vocational education, if it is conducted at all, is typically 
limited to first-year or' nontenured teachers, and commonly 
excludes the many individuals other than instructors who con- 
tribute to the quality of education. 

, In addition to the evaluatipn concerns mentioned above, a 
comprehensive 1 program evaluation must consider the cost of the 
program in relation to the short- and long-term benefits it 
produces, the program 1 s ^efficiency in producing benefits, and 
any unanticipated and perhaps unwanted effects that accompany 
the benefits of the program. 

\ Requirements included in the 1976 Education Amendments move 
the evaluation of vocational education programs from models into 
practice. Along with the VEDS* reporting requirements discussed 
earlier, states must submit planning and accountability reports 
covering the following topics: 
t 

\ Planning and operational processes including: 

(1) _jqu_ality and availability of instructional 
offerings, /4 

(2) guidance, counseling, placement, and follow-up ^ 
services, 

(3) capacity and .condition o£ facilities and equipment , 
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(4) employer participation in cooperative vocational 
programs, 

(5) teacher/pupil ratios, and » ' 

... I . 

(6) teach^^ualif icafciqias ; 

Results of student achievement a? measured by standard 
occupational proficiency measures or other methods; ' 

) * 
Results of student employment success, such as wages, 

employment and -tfnemp Lament , and ffraployer satisfac- 
tion; and ^ \ 



• Results of additional services, including service to 
special populations. 

Finally, the states 1 annual application for continued funding 
wi'll describe the vocational education needs of potential stu- 
dents and indicate how and to what extent the program proposed 
*\wil 1 x meet these needs and describe how the findings of any 
evaluations of programs have been used to develop the p.roposed 
program (Grasso, 1979). These federal evaluation requirements 
have stimulated the development of v statewide vocational educa- 
tion evaluation systems. • ; 

The 'Rale of the Curriculum Specialist 

It is not the purpose of this module to tjain evaluation 
specialists to cqnduct comprehensive program evaluations in 
vocational education. This goal is clearly beyond the scope of 
even an extensive series of modules. Preparation of a competent 
evaluation specialist requires {graduate level Qoursework, per- 
haps an advanced degree, and a substantial amount of experience 
in actually conducting program evaluations. Short of this, an 
evaluation coordinator must rely , on consultations with experts 
to fill in holes in his or her background and experience if the 
evaluation of a vocational education program is to be done as* 
proposed by: the model builders and as implied in federal 
requirements. < ^ 

Program evaluation responsibilities may fall on an indi- 
vidual curriculum specialist as part of the duties of his or 
her unique employment situation. However, the curriculum spe- 
cialist role itself does not require all the competencies of a 
program evaluation specialist. The curriculum specialist's 
major Concern in the ar6^kof evaluatiort is to collecj^data and 
to make decisions for the purpose of improving or judging the 
worth of his or her product. In this* sense, the product of a 
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curriculum- specialist is, of course, the curriculum he ol>^he 
has developed. * \ * 

Vocational Education Curriculum Evaluation 

The curriculum specialist should consider his or her pur- 
pose in conducting aij evaluation to arrive at an appropriate 
focus for his or her evaluation activities. As stated above, 
this purpose is to improve the curriculum or judge its worth . 
To accomplish. this purpose, it. is necessary to examine three 
closely interrelated areas: the planning and development of 
the curriculum, the way it is being used, and the outcomes it 
produces. 



Assessing curriculum planning and development * In assess- 
ing curriculum planning and development, the curriculum spe- 
cialist examines the steps used to; ( 

• . \ / 

• decide that t;he curriculum being evaluated wa8 ^ 

actually needed \ 

• determine its content, goals, and objectives; and 

• select learning strategies and materials. 

In short, he or she assesses the rationale for the curriculum 
and its content and structure in order to validate his or her 
product. 

The modules 'in this series that present instruction in 
curriculum planning and development can provide a fiSamework for 
deriving appropriate questions for assessing curriculum plan- 
ning and development. This area of vocational education curri- 
culum evaluation is signif icant,~~if often overlooked. If the . 
initial pfias^s of curriculum development are not well done, the 
resulting product is not likely to be worthwhile. If this is 
the case, improving the , planning and structuring of^he curri- 
culum is the most basic step in .enhancing its worth.** 



Assessing curriculum implementation . Assessing the imple- 
mentation of the curriculum is the second area of curriculum 
evaluation with which the curriculum specialist must / B"fc\c^n- 
cert\ed» The ultimate criterion in the evaluation of curriculum 
must be the outcomes that the curriculum produces in the stu- r ^ 
dents who experience it. However, before outcomes can be 
assessed, it is necessary to determine that the curriculum is 
being used as was intended. If this is. not the case, the cur- 
riculum specialist should not proceed to an outcome evaluation, 




but rather concentrate on solving implementation problems, 
•either ^by revising the curriculum or by assisting teachers to 
use it as planned. 



* Assessing, the outcomes of a curriculum . The assessment of 
the outcomes of a curriculum is the primary focus of the 
remainder of ' this module, although the other two evaluation 
arfeas with which a curriculum specialist should be concerned 
will be mentioned. The modute concentrates on evaluating cur- 
riculum outcomes because this is the aVea in which vocational 
education evaluation is most different from evaluation In other 
areas* of education^ The difference arises because the outcomes 
of vocational education can be more clearly defined. In gen- 
eral, the outcomes of a vocational education curriculum are 
students who possess the job skills nlcessary to obtain and % 
advance in an occupation, and who meet the needs of employers* 
The assessment of student learning and the attribution of that 
learning to the curriculum wil^^be^th^topic of the remainder 
of thiy module. This represents an evaluation of the short- 
term outcomes of a curriculum. Assessing the J.onger-range out-& 
comes of the curriculum through follow-up studies with students 
and employers will be the t focus of the other evaluation module 
in this series. • 



r 
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Individual Study Activities 

1. Read Chapters 1 and 2, pages 2-67 in Vtentling, T. L. ; 
Evaluating occupational education and training programs . 
Boston: Allyn and Bacon, 1980/ Read pages 246-263 of 
Chapter 11 in Finch, C. R. , and Crunkilton, J* R. , Curri - 
culum development in vocational and technical education 
Boston: Allyn and Bacon, 1980. f 

2. In a sentence or two, state the parpo^e of each of the 
three approaches to vocational education evaluation that 
are discussed in this module. 

3. Write a paragraph that explains why a curriculum special- 
ist should conduct an evaluation of a curriculum he or she 
has developed. 

4. Below is a series of questions that could be asked during 
an evaluation of a vocational education curriculum. On 
the line-in front of each, write: t 



"1" if the question is appropriate ( to an assessment of 
curriulunyplanning and development; 

!, 2 n if the .question is appropriate to assessing curri- 
culum implementation; and 

»y»-if *the question is appropriate to assessing the 

* outcomes of a curriculum. 

^. Do employers think students who experienced the 
' curriculum possess the skills necessary to p^r- 
* form work tasks adequately? 

b. X Was the film demonstrating u^cr^er work procedures 
~ available on schedule? / 

c. Are the objectivesof A he currTculum based ^n an 
" adequate job description? 

d. Is the curriculum, 'based on an accurate forecast of 
~ ' ' population and labbr market needs? 

e. Were the students 1 .scores on the performance test 
: ' up to required standards after they^completed the 

curriculum? 

* » 

f. Do students think the curriculum included pre- 
' paration in the skills they need on the job? 
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Are students receiving the recommended amount of 
instruction in the ifir.pt segment of the curri- 
culum? ; 

h. Why was the unit shop approach used to organize 
this curriculum? 

« 

i. Were the objectives of the curriculum appropriate 
to the entry-level jobs students were able to 
obtain?' * 

« 

j. Are the instructional strategies recommended for 
* use appropriate to the students who experience 
this curriculum? 



Discussion Questions 

1. Discuss the various definitions of evaluation mentioned in, 
the module and readings, and comejyp with a definition 
that is appropriate to the curriculum specialist 1 s purpose 
of improving and judging the worth of curriculum. 

* 

2. Describe examples of vocational education evaluations drawn 
from class members 1 experience and discuss which approach 
or approaches to evaluation each example illustrates. 

3. Discuss the effect of federal vocational education evalua- 
tion requirements on the development of evaluation strate- 
gies, procedures, and techniques. 



Is it possible to do a "comprehensive program evaluation 1 



5. How is an evaluation conducted to improve a curriculum 
different from one conducted for the purpose of judging 

^ the worth of a- curriculum? 

6. Discuss the interrelationship between the three areas of 
vocational education curriculum evaluation: assessing 
curriculum planning and development; assessing curriculum 
implementation; and assessing the outcomes of a curriculum. 



Group Activity . * j 

c 

Collect at least three instruments such 'as tes^s, question- 
naires, checklists, etcJtixat have been used in vocational edu- 
cation evaluations. Consider^in which evaluation approach(es) 
(accreditation, accountability, or comprehensive) each might be 
used. Consider whether each might be appropriate for use in a 
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vocational education curriculum evaluation. 



If this 'is the 



case, consider for whiqh area of vocational^ education curricu- 
lum evaluation (assessing curriculum .planning and development, 
assessing curriculum implementation, or assessing the outcomes 
of a curriculum) each instrument migTit be used. Examine each 
item on each instrument. For what types of decisions might • 
each item supply information? / 
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GOAL 2: Describe the purposes and major activities involved in 

evaluating-a vocational education curriculum prior to 
1 its implementation* . 



Evaluating Curriculum Before Implementation 



This module will now addresrf t;he concepts and procedures 
involved in evaluating the short-terra outcomes of vocational 
education curricula. Two primary phases of evaluation will be 
discussed:, the pre-implementation *and *the implementation 
phases. Pre-implementation evaluation activities are normally 
carried out during the cToh^eptualization and early development 
of a curriculum, before it is ready to be. used with its intended 
audience. Their purpose, obviously, is to help improve the 
>i4Ximate curriculum. 

Implementation evaluation activit£?s? on the other hand, . 
are applied to the operational curriculum, after it has under- 
gone at least a minimum of ttial arid revision at the pre-imple- 
mentation phase. Their purpose is to assess the effectiveness 
of the curriculum in attaining intended outcdmesUnd to facili- 
tate decisions regarding the future of the curriculum itself. 

There is no simple rule dictating when the implementation 
evaluation should begin; in general, however, it should not 
precede a demonstration that the curriculum is capable of pro- 
ducing its desired outcomes for its intended audience. 



Analyzing the Curriculum Before Evaluating 

/ , * 

Before 'even beginning pre-implementation evaluation 
activities, it ijs important to analyze several aspects of the 
curriluLum as they relate to -achieving desired outcomes. 



Review inCended curriculum outcomes . If the objectives of 
a curriculum are_ not worth attaining, it is trivial to measure 
how well they ari^ attained. Therefore, it' is desirable to review 
the ootcomes that are supposed to result from implementing the 
curriculum. Questions such as the following should be' asked; 
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• Is there rea$on to believe that the outcomes are 
l important and not already accomplished by the intended 

audience (i.e., is there a need for the^ product)? 

if 

• Are the desired outcomes* described specifically e'nough 
to be observed and measured (i.e., can their attain- 
ment be measured)? 

• Are the desired outcomes feasible in light of the cir- 
cumstances and appropriate for the intended audience 
(i.e., is it reasonable to fxpect the outcomes to be 
attained)? m ^ 

If the answers to any of these questions appear to be f 
negative, immediate revisions in ^the desired outcome? should be^ 
sought. 

Review logic linking proposed activities to desired out - 
comes . Regardless of the nature of the vocational education 
curriculum, it will contain a set of activities to be performed 
„Jby__an intended audience. These couLd«J.nclude_^ctAi«^--w 
demonstrations, slide or video sequences, field experiences, 
simulations and laboratory experiences, and so forth. It is 
worthwhile, at this preliminary point, to review logically the 
rationale that links the proposed activities to the desired / 
outcomes. In so doing, the following questions might be asfcfed: 

• - * « 

• Do the activities relate to the desired outcomes 
(i.e., if the activities are implemented a« planned, 
is there a reasonable chance they will' produce the 
desired outcomes)? \ 

, • Are the activities appropriate for the circumstances 
and* constraints likely to exist and for the intended 
% audience (i.e., are the activities practical)? 

• Are the activities likely to produce unintended out- 
comes that will affect *the overall impact of the 
curriculum (i.e., are there likely to.be mitigating 

side effects)? * * 

* *■ 

• Are the activities likely to be cost-ef f ective -(i. e. , 
will they be worth their cost in time, money, and 
effort)? 

As # with t/e analysis of ^tended outcomes, if the answer 
to any of these questions is negative, revisions should.be made 
and problems ^resolved before proceeding. 
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Divide curriculum i nto components, if necessar y. ^Tf a 
curriculum involves several different type^ of activities (for 
example, an individualized vocational training module, a lab- 
oratdry experience, and a simulation situation),, it^is wise to 
divide them up and try them out separately. This also' requires 
identifying and* reviewing desired outcomes .for each component, ' 
a|d analyzing the link between the activities of each component 
and the outcjpmes. 



.Reanalyze logic of curr iculum prior to conducti ng 'the 



implementation , n ipgicai analysis of the^urriculunk similar 
to that recommended prior to, implementation, is .the first step 
in evaluating the ultimate I'mpfct 'of the curriculum. Some of 
the questions asked earlier, as well as somewhat broader ques- 
tions such as the following should be posed: 

» _ t . • . 

• Axe the peeds the Curriculum is designed to meet 
important? ^ , 

• Are .the needs defined in terms of required changes in 
— — ^bG^v&y-pevpl^&n — — ^™ 



D6 the desired outcomes of the curriculum relate to 
the needs it was designed to fill?. Is, there a reason- 
able chance that if the outcomes are attained, the . 
leeds will ,be, met? 



• Are the desired outcomes of the curriculum appropriate 
^for its intended audience and circumstances of use? 

• ^Are th£ activities likely to produce the desired-out- 
comes under the intended circumstanfces of use? 

• / Are the activities 'implementable within practical time 
And resource constraints? 
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Are there likely to be, no unintended side effects, 
jf particularly undesirable ones? 

Are ihe effects of the curriculum measurable in the 
short run? If not* can crther suitable measures be 
deve loped? 



I|f the pre-implementation evaluation was done* properly , 
the alcove questions will likely be answered "yes," and a full- 
scale evaluation can begin. Negative answers, at this point in 
tjjie, /require clarification at .the very least and possibly more * 
substantial curriculum revisions before proceeding. 
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Pre-implementation Evaluation 

As previously stated, it is desirable to subject early ver- 
sions of a vocational education curriculum to empirical tryouts, 
/ or pilot tests. Such' tryouts. should represent an attempt to use 
' the curriculum (or its components) on a small scale under con- 
trolled circumstances in order to gain a's much information as 
possible to guide subsequent revisions. These tryouts should 
be able to yield data that permit immediate revisions to facili 
tate larger-scale implementation of the curriculum. 

Pilot testing does require resouWe$ (time, money, person-* 

* A T '- ^1 T¥hari!Ttfrr^TW^ 

tations might, also preWt ottiife tW^commended revisions. 
If resources are thus limited, it may be] necessary to implement 
the curriculum without the benefits of tryouts.. It is impor 
tant to-We, however, that the quality of a curriculum can 
almost always be improve* by devoting even minimal resources to 
preliminary tryouts. The omission of these activities entirely 
from any curriculum development effort is unwise. 

There are, however, occasions when tryouts are no* needed. 
If-an activity is a one-shot event that will not be repeated, 
there would be little need for a tryout. An example of this 
mieht be use of a regularly-scheduled TV program on new para- 
^essional occupations in the allied health fields, in which 
the program was available for viewing only once. 

& . Similarly, activities alreadf In their final form for 

* which there is no intent to review should not be pilot tested. 

These normally would include traditional activities whose con- 
tent depends on precedent rather than on intent to produce 
denned outcomes. An example of this might be an orientation 
. for an incoming electronics training class. 

It is often impossible or impractical to conduct tryouts^ 
on every component of a curriculum s ilnce resources are invari- 
ably limited. ActivitieS that should gettop priority are 
#*hose most crucial to the success of the entire vocational edu- 
* cation curriculum and about which there is the most uncertainty 
in terms of achieving desired outcomes. 

In deciding whether to pilot test-, itmight be useful to 
rate the activities selected for implemen^ion cm the four 
scales shown in the figure that follows. - 



V 



-32- 

27 



REP LICABILITY 
~h 



is ^no^Tht 



There is no^Thtent 
to repeat this activity. 



If this activity proves 

essful, it will be 
used repeatedly in the 
future. 



REUSABILITY 



The structure of this 
activity does not require 
further improvements. 



There is a strong 
desire to modify and 
improve this activity. 



IMPORTANCE 



The success of this 
activity is not crucial 
to the success of the 
vocational curriculum. 



TTfe success of this 
activity -is absolutely 
essential^to the suc- 
cess of the vocational 
curriculum. 



UNCERTAINTY 



There is no question 
about^ the potential 
effecte^of this activity. 



There is considerable 
cfoubt about the poten- 
tial effects of this 
activity. 



Sum tile ratings and rank order the sums. Activities that 
score highest should receive whatever time and financial 
resources are available for pilot testing. 

I 

The most important information to be gained from pilot test- 
ing is whether the curriculum^C9 m P onents selected are^ capable of 
producing the desired outcomes in its intende'd audience. 
Acquiring this information through empirical tryouts involves • 
several sequential activities: .(1) planning the tryouts; 
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(2) developing outcome measures; (3) selecting an evaluation 
design and sample; (4) conducting the tryouts; an* (5) process- 
ing and analyzing the results. 

Planning the tryouts . Outlining a general strategy and 
detailing the tasks involved and time required are good first 
steps in conducting tryouts. Normally, early tryouts should be 
low cost, inv olving few students and brief time schedules ; 
these limitations, of course, are relative to the scale of the 
tasks to be performed. The general strategy is to conduct fche 
tryouts under well-controlled circumstances, ^.using members of 
the target population, collecting objective information on 
degree of 'ffttrainmeTit'trf "intemtfed * outcome^' as^weil as^any other- 
information likely to be helpful in improving the curriculum 1 . 
It is important to avoid collecting too much data at this 
stage, since the scope of revisions will probably be limited, 
and since data processing and analysis will have to facilitate 
immediate feedback." 

The focus of the tryout is to obtain data regarding the 
attainment of student outcomes. However* it is also desirable 
to plan to obtain information on the degree to which the 
planned activities are being carried out as intended, so that 
any necessary refinement's in the process of teaching can be 
made, and so that ultimately the outcomes can be attributed to 
proper implementation of the curriculum. 

In addition, plans for gathering data on unanticipated 
side effects, both positive and negative, should be made so 
that the positive ones can be enhanced and the negative ones . 
lessened. Take this hypothetical example: 

Pretest-posttest tryout data on the knowledge of 
concepts taught by a film on innovative techniques in 
computer programming showed a sizeable increase in 
knowledge among 90% of those tested. The evaluators 
were pleased and decided to use the film with no 
revisions until one female participant remarked that 
the film's sexist language offended her. Subsequent 
review of her claim led to substantial script revi- A 
sions. Without her chance remark, the film might 
well have produced negative side effects in about 
half of its intended audience. 

Developing process and outcome measures . Outcome measures 
are developed to provide a quantitative description of the 
extent to which an educational outcome has been obtained. Pro- 
cess measures are developed to describe how an educational out- 
come has or has not been obtained. Much has been written on 
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these subjects, but it is not the intent of this module to pro- 
vide a course in educational measurement. A few basic concepts 
and techniques, however, will help assure credible measures in 
both this tryout phase and the later implementation phase. 

Measurement techniques will differ according to the type 
and specificity of the object iv es to which they apply. ^For 
example, process objectives afe^usually so direct and precise 
that measuring their attainment may involve a simple checklist 
approach such as "Yes, it was done, 11 M Nof^ it wasn't" items. 



Example: * * 

Check one 

The instructor demonstrated proper and Yes No 
improper safety procedures in the use 
of a table saw. 



Similarly, a student performance objective that calls for 
the carpentry "student to. identify three types and uses of power 
$aws wpuld involve a fairly straightforward questioning approach. 
On the other hand, an objective requiring the sales student to 
establish 'good rapport with a customer would be more difficult 
to measure. That some outcomes are easier to measure than 
others does not mean that trivial, easier-to~measure'objectives 
should be substituted for more -important^ harder-to-measure out- 
comes. If arr outcome describes something someone ctan do, say, 
think, or feel, it is measurable. *J 

Although there are numerous measurement techniques avail- 
able, mastery of a few basic techniques will allow the evalua- 
tion of most vocational education outcomes at both the tryout 
and implementation stages. Two basjLc categories are applicable: 

• Written tests (paper-and-pencil instruments). These 
include true-false, multiple-choice, completion, and -similar 
objectively scored test quest ions ; essays ; checklists; rating 
s'cales; and questionnaires. Generally, these pose fixed ques- 
tions and require a written response. They are easy to adminis- 
ter and score and can easily measure both knowledge and attitu- 
dinal objectives* , / 

, • performance tests . Performance tests are vitally , 
import-ant in vocational education. Such tests pose a fixed 
question ot situation and require the student to do and/or 
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produce something. The Response is observed and scored accord- 
ing to predetermined standards. In vocational education, per- 
formance tests take two forms: (1) the performance test 
requiring the student to -accomplish a job-like task under con- 
trolled and observed conditions, and (2) the product evaluation 
wherein the product resulting from the performance is evaluated. 
(The latter, in contrast to the former, does not allow for 
determining whether the correct process was performed.) Further 
X v information on types of tests and their applications in voca- 

,\ ) tional education evaluation may be found in the module ^n this 
series that deals with the selection of instructional s^-ate- 
gies and the assessment of student achievement. 
* 

* "Si'rf£2 Small^scsle 'tryoots arrer 1-ike-ly -to-empioy 'measures of * 
outcomes constructed by the evaluator, it is crucial for him or 
her to be aware of and to apply conscientiously certain well- 
accepted considerations that help to ensure adequacy of mea- 
surement. These considerations are listed below, and will be 
discussed in detail later. 

• Objectivity . Will the technique yield the same score 
regardless of who is applying it? 

• Reliability . Does the technique produce data that are 
free from random error and thus yield a relatively , 
constant score? 

• Validity . Does the technique measure what it is sup- 
m y / ^ posed to be measuring?' 

* • Efficiency . Is the technique relatively cheap and 
easy to administer? (This is especially important 
during tryouts.) 

• Non-reactivity . Does the technique unduly influence 
the subsequent behavior of the respondent? 



Selecting a sample and an evaluation design . In conduct- 
ing tryouts, the curriculum specialist is most interested in 
inferring from the results the likely outcomes of larger-scale 
implementation, and in making revisions to improve attainment 
of outcomes. i Thus, it is necessary to select persona for the 
tryout who are at least broadly representative of tfie intended 
audience. In practice, this often means identifying five or 
six persons who are likely to do very well, very poorly, and 
Average on the activity. Strict random sampling procedures ^ 
* (wherein each student has an equal chance of being selected) 
are rareiy used because small random samples are not likely to 
be broadly representative of the audience. 
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The terra "evaluation design" r£*ers to the arrangement of 
persons, an activity, and outcome measures used so that infer- 
ences can be made about the probable effects of the activity on 
a larger group of similar persons. Although design considera- 
tions arise more often and are more complex in conjunction with 
evaluation after implementation, a word on design for tryouts 
'is in order. 

The design recommended for mosrnrfjrout appiications^TFTRe' 
"one group pretest-posttest design. 11 In this method, outcome 
data are gathered both before and after the curriculum has been 
tried out. In using this design, it is especially important 
that the measures used do not ip- themselves unduly influence 
the 'subsequent pe^fo^rmaftice^af* the'-perOTM reefed. With appro- 
priate measures, this design will provide information on the 
degree to which students can already perform the desired out- 
comes, and the gain in pe/forraance that is probably due to the 
curriculum. Sometimes, liimited resources or circumstances may 
pre^pnt administration of^ pretest, and a posttest-only design 
will suffice. The expenses associjateti with using a control 
group preclude this type of design for a tryout. 

As a reminder, it is important at this stage also to col- 
lect process data regarding implementation of curriculum, 
so that problems in materials and procedures can be resolved as 
quickly as possible. Similarly, it is important to be on the 
lookout for unintended outcomes so that 1 revisions resulting 
frdm^such outcomes can be made. 

Finally, care snould be taken to ensure that student out- 
comes can be related to that student's pretest data and to any 
personal traits (e.g., sex, ability, level, motivation) that may 
have affected the outcomes. Observations in this area can help 
predict whether there will be differential performance effects 
during large-scale implementation. 

Conducting the tryouts . A mimber of fairly straightfor- 
ward tasks are involved in actually collecting the tryout data. 
First, the tryouts must be scheduled. The schedule should 
include, if needed, training for any test administrators, noti- 
fication of students' and instructors involved, and the testing 
itself* • 



Second, test administrators must* be selected and trained.. 
Depending on the Qutcome measures used, it may be necessary to 
exert care in selecting appropriate persons and in orienting or 
training them. These efforts, however, are likely to be low-key 
in small-scale tryouts. t * 

• * 
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Third, the identified students, and possibly staff involved 
in providing process information, shou^d/oe notified af the 
purposes and procedures involved in thexryouts. And finally, 
the data can be collected according to the plaris and j prQcedures 
outlined. • | 

- -"-^-grocggg^ and using the data . Sinc^ the num-* 

ber of persons involved in tryouts is small, the resulting data> 
can be tabulated simply. Analysis of the data should be simi- 
larly straightforward. Percentages of items answered correctly 
or checked in a particular category, or percentages^ of partici- 
pants able-^to perform- at specif ie.d. levels, wilf nctrmally ^ufr:^ < 
fice. Most important, the analysis should not hide the ability 
of the data to indicate where revisions are needed. 

A number of decisions must be made as a result of tryouts. 
Data are especially useful in helping to support the develop- 
mental hypothesis that the curriculum activities are in fact 
appropriate for the intended audience. For example,* if the 
pretest data show that participants can already perform the 
•intended outcomes, then the need for the activities becomes 
questionable. If, on the other hand, data indicate that the 
concepts introduced are totally foreign to the audience, modi- 
fication of the jlctivities or the intended outcomes might be in 
order. 

Posttest tryout data may show that desired outcomes are 
substantially beyond the capability of the curriculum to 
achieve. Such a finding might prompt major revisions or even a 
reassessment of the need fdr*the curriculum itself. More 
likely, these data will suggest specific instructional problems 
in various components of the curriculum. Then attention can be 
directed to remedying such_problems *prior *to, large-scale usage. 
Sometimes further tryouts may be needed to refine the curricu- 
lum further before full implementation. 
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Individual Stiidy Activities 



( 



1. Categorize each of the following evaluation activities 

according to whether its intent is to collect process (P) 
or outcome (0) data. Indicate P or 0 in the space pre- 
ceding the activity! ' ' 



r 

Determine whether appropriate books and materials 
are being used by students in a home economics 
class. 



b. Observe student demonstrating proper woodworking 

' c. .Evaluate the quality of a simple program devel- 
•oped by a computer programming student. 

j 

d. Interview a nursing student to determine which 

parts of a coifrse were most helpful in teaching 
intravenous techniques and applications. 

e. Administer a multiple-choice test on sales 

strategies to a retailing cltfss. 

\ 

f . Det^tjoine whether drafting teachers are following 

a prepared plan of instruction. 

\ 

g* Observe ahd_as^ess how well a home economics jstu- 

de'nt applies a zipper to a dress. 

2. See*pj>. 80-95 (measures of ^knowledge) and pp. 95-124 (mea- 
sures of performance/ in Wentling, T. L. Evaluating occu - 
pational education and training programs . Boston : Allyn 
and Bacon, 1980. 

3. Select from the list provided on the next page the type, of 
measurement technique the evaluator might best use to 
obtain answers to the following questions, and indicate it 
in the space preceding each question. 
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Measurement' techniques: 



1. Written tests (e.g., multiple-choice, true- 
false) 




Product evaluation 
Performance test 
Checklist 
Questionnaire 

a. Is the student able to remove and replace nuts 
and bolts with an air wrench properly? 

b. As a result of the industrial arts program, will 
students have an improved attitude toward good 
craftsmanship? 

c. Will the nursiag students understand the physio- 
logical reasons and treatment for chronic 

<> hepatitis? 

d. Will the business students be able to identify 
the qualities of a good employee? 

e. Are all personnel involved in the forestry pro- 
gram satisfied with the way -it is being conducted? 

f. Are all prescribed equipment and materials for a 
course in bench metalwork available to students? 

\ 

g. Are the buttonholes and fasteners on a tailored 
coat applied properly? 

h. Is the program in video electronics being imple- 
mented as planned? 

i. *Is the student able to obtain an accurate reading 
of a patient's Jblood pressure using cofrect 
procedures? N 

j; Is the business letter typed in accordance with 
standards described in the course manual? 



Discussion Questions 



a. What are some examples of everyday evaluations we all 
make? What criteria do we consider in making these 
evaluations? Why are everyday evaluations made? 



r^-Whar^are^cmm--coinmon , everyday evaluations TfiaFTT?~~~ 
, ^ vocational teacher might make? What criteria might be 

considered, and how are these criteria unique to voca- 
tional education? ' Why might these evaluations be 
conducted? ^ 

2. Discuss under what conditions it is or is not appropriate 
to conduct pre-implementation evaluation activities. What 
constraints might limit activities in this* area? Try to 
relate your discussion to examples from your own setting. 

3. What is the difference (are the differences) between 
assessing processes and outcomes in the "pre-implementation 
phase of evaluation? Why is it important to assess both? 

4. An electronics technician instructor has developed some 
new exercises to improve the manual dexterity level of 
students. -The department head is thinking about using 
these exercises throughout the department.^ What should be 
"done to try out these exercises prior to implementing them 
departments-wide? Consider such things as when to try them 
out, with whom, under what conditions; likely ways to pro- 
cess, analyze, and use results; etc. 



"Group Activities 

' / 

1. The purpose of this activity is to provide practice in 
analyzing a vocational education curriculum in terms of 
its defined student outcomes and the relationship between 
the outcomes and the proposed activities. 

Part of a new woodworking .curriculum is presented in the ' 
\^ figure ot\ the next two pages. NAssuttfe that the curriculum 
\^is adequately budgeted and that\t is to be used with new, 
^inexperienced woodworking students. -In your gr&up, analyze 
thk curriculum. Use the questions outlined intKo^ection 
on ^Analyzing the Curriculum Before Evaluating 11 (Goal 2 of 
thip module) as your discussion guide. 
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Figure 1 
Woodworking Curriculum 



Student Outcomes 



State the importance 
of the woodworking 
industry. 



2. Develop good work 
habits. 



Develop an under- 
standing of £abor 
and management. 



4. Develop an appre- 
ciation of good 
craftsmanship. 



Proposed Activities 

1. Invite students to present reports on some 
aspect of the woodworking industry, 

2. Visit several woodworking- industries, 

3. Have panel discussion abotft "Contributions 
of the woodworking industry to home and 
family." 

1. List specific standards for each job. 
Discuss. 

2. Discuss importance of good habits vis-a-vis 
industrial practices, job advancement, and 
so forth. # *" 

3. Provide a clean, organized workshop conducive 
to good learning. 

4., Provide means for students to plan their work 
methodically and accurately. 

5. Measure students' work objectively. 

* 

1. Give reading assignments in trade magazines* 

2. Arrange for visits by persons in industry 
representing both management and labor. 

3; Have discussions on problera*-<^lating to 
labor unions, responsibilities 0^ manage- 
ment, and personnel relations. 

4. Have suggestion box for ideas about running 
a wood shop. 

I. Arrange field trips through at least two 

factories, one that produces cheap furniture 
and one that produces fine furniture. 

2* Have students analyze products for good and 
poor construction, design, etc. 

3. Discuss principles of good design and good 
workmanship. 

4. Hold a contest to display projects -of good 
craftsmanship and design. 
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2/ Read each of the following situations* Discuss each one 
in terms erf the "lesson. 11 it teaches regarding desirable 
/ pre-implementation evaluation strategies. In other words, 

A what might you do differently in a similar situation to 

improve the evaluation effort? a % 



In May, a team of developers at ADS, Inc. was giver a 
contract 'to prepare an innovative curriculum in adver- 
tisinj^aijd 'sales for the Fountainville Community Col- 
legffDistrict , on t;he understanding that the developers 
would be able to deliver the following March. The team 
members got to work at once, but since summer was over 
by 'the time they finished the materials aod developed 
the accompanying book and AV materials, the team had 
everything produced and began a full field test in 
August without" bothering to try the components out - 
first. When results were finally in, the field test 
revealed a number of serious problems. The revisions 
required were so extensive that the materials could 
not b,e completed by the deadline, and it was therefore 
the following autumn before the program could actually 
be implemented in the community colleges. , 

Another team at ADS, Inc. had developed a business 
training program. with a short film illustrating several 
types of employee-employer interactions. The team had 
carefully prepared a pre-posttest to measure learning, 
and the film was shown to a small number of persons 
from the intended audience^^men the tests were 
administered and scored, tne team found that jnost of 
the subjects, while performing at about the "chance" 
level on the pretest, got about two-thirds of the 
items right on the £osttest. They were still discus- 
sing the implications of this when one of the team 
happened to hear a student remark, "You know, that 
film was a nice idea, but we all thought the narration 
was a bore. 11 

A home economics textbook writer recently explained 
that he did indeed try out his'book before publica- 
tion. He had several teachers use it jLn their class-* 
rooms for a semester and then report their reactions. 
When- asked what aspects of the book he was interested 
in evaluating, he replied, "None in particular. I was 
interested in -reactions in general." 

Stan Binet was monitoring the tr-yout of a self-instruc- 
tional booklet intended to help technical education 
teachers improve thfcir ability to 'Construct tests. 
Since it was summer vacation, he had been able to find> 
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a group of experienced teachers to use as subjects, 
and for convenience he had asked them to come to his 
office and work in a room set aside for that purpose. 
When the tryout began, St&n found ■ that the teachers — 
spent a lot of time drinking coffee and talking to 
^ each other, and their comments on the materials seemed « 
to be a* consensus rather than a collection of indi- 
vidual reactions* After he inadvertently mentioned 
that he had helped write the booklet, the severity of 
their criticism decreased noticeably. 4 

! f 

e. Strategy Simulation Games, Inc. was working on a simu- 
lation exercise to improve the safety habits of indus*- / 
trial arts students. The game was tried out with five 
groups of students of different ability levels, and ^r^.^ 
trained abservers noted the behaviors of each student %' 
ate he or she performed the exercise. The observers 
accumulated five folders full of new data, in addition 
to pre- and posttests on safety habit levels. When 
the tryout was Over, the data wer$ delivered to the 
staff member assigned to review the exercise. .She - 
spent a month analyzing and introspecting about the 
datte and produced a 79-page pilot test report. The 
report arrived in 'the Strategy Simulation Games, Inc. 
offices top late for the exercise to be incorporated 
in' the industrial arts curriculum for which it was 
designed. ' * 

These situations were presented to illustrate some of the 
general principles aijxi^strategies of pre-implementation 
evaluation stressed in the module. In your group, discuss 
these principles and derive a general strategy for such an 
evaluation. Discuss such questions^ as the following: 

• Why is it important to pilot test? 

• # What should be evWffated? 

• What types of activities should be involved? 

• What constraints might limit your efforts? 

• What controls should be' placed on the efforts? 
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GOAL 3: 



V 



Describe th0 purposes and major activities involved in 
evaluating a vocational education curriculum after it 
has been implemented* 




Evaluating Curriculum After Implementation 



j Whether v or not a curriculum specialist is accountable to 
s /?5^9J£tside authority for the effectiveness of his or her cur- 
riculum, potential users need to have some assurance that it 
works/. Thus, an evaluation of the curriculum 1 s performance 
> should be carried out soon after it is implemented on a rela- 
tively large scale. Such an evaluation attempts to answer 
questions like these: j 

' • Can the curriculum be implemented by intended users 

who are not under the direct control of the developer? 

/" 

• v Is th£ curriculum successful in producing its desired 
^[outcomes? Are these produced with a minimum of unde- 
sirable side effects with all members of the intended 
audience?" 

• Are desired outcomes produced under a wide array of 
circumstances and situations* (i.e. , how generalizable 
is the evidence of effects)? 

• Are desired outcomes produced within acceptable cost 
limits and likely implementation constraints? 

• Ate the outcomes stable over time? 



Implement at ion*Evaluat ion Concepts 

The concepts and procedures of an implementation evaluation 
are somewhat more detailed than those already presented for pre- 
implementation tryouts. But they are^basically the same, 
involving, after the initial logical analysis of the curricu- 
lum, the following steps: 

9 




• Planning the evaluation rw * / 

• Developing process and outcome measure 

• Selecting a design and sample 

. . v • i 

* • Conducting the tests : , 

• Processing, and analyzing tfte results ^ x 

Since many of the concepts are so similar to those already dis- 
cussed, an attempt will be made in the following sections to 
avoid repetition and to elaborate on aspects unique to this 
evaluation phase. 1 

* 

Planning the evaluation . There is no one "right" time to 
conduct the evaluation. The ^timing depends largely on the, pur- 
poses of the evaluation. When program personnel need informa- 
tion to make major decisions, an evaluation is in order. How- 
ever, it is best to wait until the program is well underway 
before attempting an evaluation. ^ 

4 r 

Obviously, it is necessary to know which program personnel 
n£ed information and what information is needed. What ques- 
tions must be answered? Some of the likeLy questions have just 
been listed on the preceding page. 

"** is v , In addition, the evaluator must know what resources are 
available to conduct the evaluation so that practical limita- 
tions can ,t> e considered in planning. Such resources include 
money; professional, clerical, and other labor resources avail- 
able; supplies; computer facilities; and so forth. 

Because resources are never unlimited, it will be neces- 
sary to select the outcomes *to be 'evaluated. The choice will 
naturally depend on the decisions that need to be mad^. An 
effort should be made »to meet these decision needs rather thajj 
to focus only on the easy-to-measure outcomes. It is better to 
have limited data on important outcomes than to hav^ lots of 
•data op trivial ones. J 

In addition to selecting the outcomes for measurement, 
possible positive or negative side effects should be identified. 
Clues may be available from earlier tryout Efforts. Also, as 
in the tryoiits, it is. important to plan to gather data on pro- 
cess—that is, on the circumstances and activities associated 
with implementation of the curriculum. Process measures would 
yield 'information regarding the following: circumstances of 
implementation (staff characteristics ; physical and* demographic 
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characteristics of the evaluation site; components of curricu-: 
lum implemented and how; extenuating circumstances; atmosphere; 
etc.); costs of implementation^taf f ; equipment and materials; 
facilities; miscellaneous); and characteristics of the partici- 
pants (number; demographic traits; motivating conditions). 

~ , Collection of such process data fulfills several important 
functions. It helps to ^ensure* that the curriculum was imple-* 
mented as intended so that the outcomes measured can legiti- 
mately be related to the curriculum. In helping to describe 
the curriculum implementation adequately, it helps future users 
to implement it properly. And finally, it provides an 
empirical basis by which future curriculum developers can 
predict the effects of specified kinds of instructional 
activities on specified audiences under specified conditions* 



Developing process and outcome measures . The types of 
measures likely to be used in the evaluation of a vocational 
education curriculum have already been discussed at some length 
with regard tctryouts. Similar techniques are used during 
this phase of evaluationj^^in addition, the major £teps to 
follow in developing measures are to: 

• select the types of instruments needed to measure the 
outcomes and processes identified; 

• draft appropriate measures; 

• try out the drafted items on a small group of students 
similar in characteristics to the target group (a 
mini-pilot test); and finally 

• revis$and refine theP measures . 

In constructing post-implementation evaluation measures, 
it is particularly important -to apply the considerations of 
adequacy (criteria) listed earlier. It^should be mentioned 
that all measurement techniques have some deficits in all of 
the following criteria; what is important in designing measures / 
is to maximize the levels oi/ adequacy. Further discussion of 
how to accomplish this is warranted. 

• Objectivity . Objective methods are those that yield r 
similar scores no matter who is doing the scoring. In general, 
pafljr-and-pencil tests, checklists, and rating scales are more 
ob|jE£tive than performance tests and observations. To improve 
objectivity, it is necessary to establish scoring rules that 
facilitate, clear assignment of scores' to each response. For a 
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less objective test (e.g., an essay), -this may require develop- 
ing a key that gives sooring rules and examples of typical 
responses and their proper scores. In addition^ identifying 
information should be* removed from each response so the scorer 
will not know whose work it is. 

• Reliability . Reliable measures are 'those that yield 
constant scores relatively free from chance variation over 
time. Lack of objectivity as well as twick questions or inac- 
curate recording devices (e.g., inaccurate test keys, unwound 

.timers) can produce low reliability. To improve reliability, 
it is important that instructions and testing conditions be the 
same for all persons; that practice, or sample items, be given 
if possible to avoid effects from unf amiliarity with the type 
of measure being used; and that several measures for the same 
objective be used rather than one. 

4 • Validity . Valid measures are those that are closely 
related to and broadly representative of the outcome being 
measured. Measurement techniques that are relatively objective 
and reliable may also be relatively valid. However ,* additional 
assurance of a measure's validity should be obtained by "con- 
structing a logical rationale for each measure used (to see 
better its relationship with the desired outcome) and by pro- 
viding sufficient measures of each important outcome. An axiom 
of evaluation is that if several independent measures of the 
same outcome produce highly similar results, the measures are 
likely to be acceptably valid. A 

• Efficiency . Efficient measures are those that yield 
reliable and valid^scores at a low cost in terms of money, per- 
sonnel, and* time. In general, this means that the measures can 
be administered to groups, on a single occasion, and under 
normal rather than contrived circumstances (e.g., an ordinary 
classroom setting)*. Measures that can be scored and processed 
quickly and easily (e.g., by a clerk or machine with a simple 
key) are more efficient than those requiring more time and 
expertise (e.g.., .analyzing the quality of* a dental bridge)/ 

• Non-reactivity . A non-reactive measure is one that, 
does not unduly influence the behavior of the person to whom it 
is being applied. The classic example of a highly reactive 
measure is uprooting a seedling daily to measure its growth. 
Relatively non-reactive measures incjltide routinely collected 
records and observations — for example, observations of time to 
complete an activity and frequency of certain behaviors. Such 
techniques may not be as relevant to vocational education mea- 
surement as to othet areas, but it id" important to be aware of 
the level of reactivity and to try to minimize it. A way to 
promote non-reactivity that is likely to be relevant ih the 
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area of performance testing is to use observers who are familiar 
to students. or tq pl*ce observers so as not to intrude on the 
situation being observed. ' ? 

Selecting an evaluation design and sample * As discussed 
earlier^ an evaluation design is the arrangement of persons, an 
activity, and measures of outcomes used to facilitate inferences 
about the likely effects of the- activity on a larger group ot 9 
similar. persons. In the context of implementation evaluation, 
M the persons" are a representative sample of the defined audi- 
ence; "the activity" is the controlled implementation of the 
curriculum; "the measures of Atcorae^' are just that, intended 
outcome measures; and M the la^rr group 1 ' is the entire defined 
audience for the curriculum. The desired inferences relate to 
the five basic evaluation questions outlined at the beginning 
of this section. 

The essential element of a good evaluation design is its 
ability to support strong inferences, particularly regarding 
assertions that the curriculum as implemented caused the effects 
measured, and that the participants are a representative sample 
of the entirjfe defifted audience.^ The design that will best sup- 
port strong inferences is one in* which two subsamples of parti- 
cipants are randomly selected :/ one subsample (the experimental 
group) participates in tha curriculium implementation; the other 
(the control or comparison* gr@up) does not. In addition*, out- ' 
come measures are administered both before (pretest) and afcter. 
(posttest) the curriculum is implemented. When outcome raea- * 
sures are processed, the performance of the comparison group is 
used to indicate what the participants 1 scores would have been 
without having beeft exposed to the curriculum. 

Unfortunately, it is oftfen not "possible to use this ran-? 
domized pretest-posttest control group design properly. There 
may not be enough persons available to construct two groups. 
Available persons might not be truly representative of the 
intended audience*, threatening the generalizatjility of the 
results. It may not be possible to collect al>**che needed data 
from the non-participants who have no incentfve to participate 
in the evaluation. Under such -circumstances, the evaluator 
might have to compromise. For example, it may be necessary to - 
use a nonrandomly selected group of non-participants, chosen by 
their willingness tcr cooperate. In this case, an effort should, 
be made to provide evidence (by comparing the pretest perfor- 
mance gt the two groups) that the two groups are not inherently 
different. Remember, the strength of the inference depends 
directly on the strength of the evidence provided. 



. The strength of the inference is also directly related to 
the number off persons involved in the study. The more partici- 
pants, the greater the chances of detecting real differences 
between experimental and comparison groups and the more gener- 
alizable the results. However, it is often logistically diffi- 
cult to involve large samples in both the curriculum implemen- 
tation and the evaluation* Thus* in general, 25-40* persons per 
group are probably adequate; The groups should be kept about 
the same size to strengthen resulting statistics. If a number 
of persons drop out after the pretest, the pretest scores of the 
dropouts and non-dropouts should be compared to ensure that no 
significant factors (e.g.^, .ability level) caused the attrition. 

Conducting the evaluation . Procedures for conducting the 
' evaluation are somewhat more detailed than those involved in 
•the tryout phase but are essentially the same: scheduling, 
identifying and training administrators, identifying and ori-* 
enting participants, and administering instruments. ~~ 

A detailed schedule for training administrators, orienting 
students, ^and collecting data should -be prepared. It is best 
to collect data in as short a time period as possible to mini- 
mize disruption of regular school schedules. Rooms of adequate 
size and appropriate facilities should be reserved. Schedules 
should be made available to all participants to facilitate 
coordination efforts. ^ r 

Different types of people might serve as test administra- 
tors. Regular teachers can probably be used if the evaluation 
^focuses on paper-and-pencil tests. If other types of measures 
are used, such as performance tests or work samples, it might 
be preferable to hire outside persons from the relevant voca- 
tional area. , 

Depending on the tasks to be performed, some training of 
administrators might be needed. Training can employ a combina- 
tion of approaches including written materials, workshops, and 
practice exercises. 

Participating students may require some orientation to the 
purposes and procedures involved in the evaluation. Orientation 
should be uniform for all students and Can include"* b6th written 
and orally presented^materials. Through orientation, not only 
is information provided but- cooperat ion is *lso promoted- 

■ ; i . ' . 

' Processing and analyzing the data . The choice of appro- 
priate processing and^fwJ^sis procedures depends largely on 
the measures used, fctfe evaluation design, and the resources 
available. \ 
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It is possible that, as in tryouts, much of the data pro- 
cessing will be done manually, especially^ when samples a-re small 
and resources limited. Often, hand-scoring involves making 
judgments about the adequacy of a response or translating a 
complex response into a single score. This would be true, for 
example, in evaluating a work sample (e.g., mitering a table 
' corner; composing an adequate business letter; developing an 
efficient computer program; repairing a defective automotive 
part), v 

Where such judgment is needed, several points should be 
considered. First, does the scorer understahd the criteria and 
apply them consistently over time? Second, has objectivity been 
promoted by concealing identifying information from scorers? • 
, Third, would two or^more scorers assess the same response simi- 
larly? To answer these questions, it is desirable to sccpre a 
random sample of responses twice. If results are inconsistent, 
it would be good to retrain the scorers and rescore troublesome 
items. 

/ In maii^-^ses, a computer can be very helpful if resources 
permit* The evaluator's job in this case is to have data* 
recorded onto computer cards or tape and to determine what 
information the computer should provide and* how it should be 
presented. A computer programmer can then writa or select an 
appropriate program, and the computer will provide the results. 

Sometimes using a computer may not be necessary, but less 
sophisticated automatic data processing (ADP) equipment may be 
helpful, tme of these is a card sorter. Data are keypunched 
onto cards, and the sorter tabulates the cards numerically or 
alphabetically. If available, a card sorter ckn be particu- 
larly useful in organizing the data — for example, into specific 
groups of people, into alphabetical order for ease of locating 
'data, or into consecutive cpde numbers when identities are con- 
^cyealed. It is, in fact, often important to divide the data 
into groups according to distinguishing participant or imple- 
/ mentation characteristics. / 

c 

Lt is important to note that the ease and convenience of y 
computer processing vary with the task. One can experience 
colossal headaches with computers and ADP when simpler manual 
procedures would- suffice. 

The next step is analyzing the processed data. Instruction 
in sophisticated data analysis is beyond the scape of this 
module. Rather, simple methods of data presentation will be 
discussed, and use of more complex analysis techniques will be 
mentioned briefly. 



-53- , 

46 * 



ERLC 



Descriptive data may be presented in tabular, statistical, 
or graphic form. An example should help clarify the differ- 
ences among these methods.- Suppose that a performance test was 
administered to lpO business students as part of an evaluation 
of a curriculum component focused on 10 office machine skills 
performance objectives: 

• The degree to which -students achieved the objectives 
would 'be presented in tabular form as in Table A on 
the following page. It describes the number of objec- 
tives achieved by all students. 

• These numbers are also transformed into the percen- 
tages of students achieving each objective. These 
percentages are a type of statistical summary of the 

* data. Proportions., ratios, and averages may be used 
also. 

• The bottom of Table A presents the same data in 
graphic form. Graphs tend to be easier to understand 
than other modes of presentation. 

To facilitate decisions based on the evaluation data, it 
is often necessary to perform more complex analyses. Evaluation 
analyses conducted fairly routinely include the following: 
tests of statistical significance to determine whether differ- 
ences among groups tested are greater than those that would 
occur by chance alone; correlation techniques to help indicate 
whether the curriculum works better (or worse) under certain 
circumstances or with certain types of persons; estimating 
reliability of measurement to assess the constancy of the mea- 
sure used; and statistical gain analysis to determine whether 
posttest scores are significantly better than pretest scores 
for a particular group; 

As stated earlier, 'there is neither time nor space here to 
convey even an elementary. understanding of the statistical 
techniques used in many desired data analyses. At this point, 
the evaluator will need to gain further knowledge in this area, 
or employ the help of persons with appropriate expertise. r 

Once the data have beer\ analyzed, they must be reported 
and utilized. These steps are covered in the module in this 
series that deals with conducting follow-up studies and com- 
municating evaluation results. 
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Number and Percent of Students Achieving 
„ Objectives on an Office Machine Skills Tes.t 
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100 
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Graphic Representation of the Percent of 
Students Achieving Objectives on 
the Office Machine -Skills Test 
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Individual Study ACTlvities 



T 



1. * Imagine the following situation* and discuss what kind of 
evaluation design could be used to determine : 

* • • if the curriculum -©eeds changing^ * 

• which simulator is best, and 

• if a simulator is* needed at all. 



Three years ago, Ed Ames prepared a curriculum jfor a large 
electronics technology, associate degree program at a local 
community college. In talking to the employers -of many of 
- tl^program's graduates, he finds th*C the employers are 
not satisfied with the graduates 1 troubleshooting perfor- 
mance. Since the electronics technology curriculum is 
heavily loaded with classroom and laboratory troubleshoot- 
ing experiences with a focus on both principles and' appli- 
cations, Mr. Ames becomes confused and gather concerned. 
The instructors in the electronics tec^logy program are 
well qualified and seem to have done a good job in other- 
wise preparing theif students. 

Ttie electronics instructors put their heads together /tid 
decide that the program lacks a high-quality, easily-pro- 
grammed trouble-shooting simulator. There are three simu 
lators on the market, one by the Do-All company for 
$7,000; one by the Do-Some company for $4,300; and one by 
the Do-Little company for $1,500. The four instructors 
don't know which of the simulators would be best; the Dean 
of Vocational Education wonders if the problem is lack of 
simulator or lack of good instruction. 

there are two problems, then: 



which of the simulators is best for the situation 
(the instructors 1 problem), and 

is the problem lack of good instruction or lack of 
a simulator (the dean's problem)? You, as 
Mr. Ames, are called in to assist the dean and the 
instructors with their problems. Assume that all 
three simulators can be obtained on loan from the 
manufacturers for one year for field test pur- 
poses, and assume that twelve classes at the 
college emphasize troubleshooting. 
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2. Collect two instruments that have been used in some sort 
of vocational education evaluation activity in your set- 
ting (preferably one written test and one performance 
test). State hp^ each of the five criteria qf* adequacy 
<rah be'mer ("or ii^>roved)' -in each^test. 

3. This module deals rather briefly with statistical prin- 
ciples and techniques used in evaluations. If you feel 
you need more information in this area, read the relevant 
chapters in Weinberg, G. H. , and Schumaker^ J. A. Statis - 
tics: An intuitive approach (3i:d ed.*). Belmont, CA: 
Wadsworth, 1974. 

4. Consider the possibility of a curriculum evaluation in 
your own setting, or of a vocational education curriculum 
with which you are familiar: 

• What resources are available for evaluation? Consider 
each step of an evaluation in terms of available 
money , staff , time , and facilities . 

• What are some specific types of cfeta that might be 
collected in the evaluation? 

• Locate a person in or associated with your institution 
who has expertise in data processing and analysis, 
evaluation, and/or computers. Discuss witt\ that, per- 
son how you could best process and analyze the kinds 
of data you listed in the item above to produce mean- 
ingful eva-Iuation results. 



Discussion Questions ' 

1. ^ Discuss the following statement: It is better Jto have 

limited data dn important outcomes than extensive data on 
trivial ones. ' * 

2. Why is it important to obtain information regarding how a 
vocational education curriculum was implemented and 
whether it produced unintended side effects? 

3. Do you agree or disagree? "It is impossible to w expect 
evaluation measures to meet all criteria of adequacy. It 
is better to have inadequate measures than to have no 
measures at all. 11 

4« The desirability of randomly assigning students to control 
and experimental groups w^s, discussed in the module* What 
problems, if any, might this cause to a' vocational educa- 
tion evaluator? 
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5. What factors should be considered when determining how 
• evaluati^ald/ta should be processed and analyzed? 



Group Activities • 

1. If possible, get together with one or tw^ other persons 
from your own institution to plan an implementation 
evaluation. Using your answers to the last item in Group 
Activity 1 above (types of decisions you need to make) as 
a guide, discuss the following questions and write down 
your evaluation plans. 

' • What most needs evaluating? KFocus your plan around 
*• this.) 

• What resources (people, time, money) will likely be 
available to you? 

• Is a design needld for this evaluation? Why or why 

not? If "yes*" what d es ig n is best? WiU P ract i cal 
constraints make this impossible? What design will 

you use? 

• What will be the target population, and how will the 
sample(s)J?e selected? 

• * What objectives will you measure? 

For persons who are not currently working in an appropri- 
ate setting, use the following simulation example as a 
basis for an evaluation plan. 



Simulation 

» 

Imagine that you are the Director of New Curricula 
for the Sandhill Community College District, a district 
made up of four iarge community colleges. Your dis- . 
trict is an unusually well funded one, supported by a 
fairly homogeneous upper-middle-class constituency that 
is' closely involved ' with community col lege matters. 
Consequently, substantial resources over the past years 
have been devoted to developing new vocational curri- 
' cula for which particular community needs were shown 
to exist.. One of the new curricula developed was 
Travel Careers, a cutriculum consisting of a series of 
seven courses that focus on three major. areas: sales 
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practices, domestic travel and ticketing, and interna- 
tional travel and ticketing. The learning emphasis in 
all units is on the acquisition of knowledge and prac- 
tical skills that^ will directly prepare students to 
obtain and succeed in jobs in the travel area. 

t v 

All seven courses have been intensively pilot 
tested at qfne of the district's colleges during the 
last two years with very promising results. Final 
reVisionshave been made, and you plan to 'implement 

* the curriculum on a larger scale at two community 

, colleges next year. 

Of course, you will want to know how effective the 
curriculum will be in meeting its multiple objectives. 
Although you've had ample funds in the past for curri- 
culum development, your budget for evaluatibn is more 
limited. You can, however, count on substantial assis- 
tance f from the teaching staff and from your advisory 
panel, which is ,made up of volunteer professionals 
from local trave-1 agencies. 

Your task is now to design an appropriate curricu- 
lum evaluation, considering the variables described 
above. * . 

Get together with one or two other "Directors of 
New Curricula" ta design an evaluation, using the 
questions listed above as a guide. 



2. The purpose of this activity is to help you develop an 
understanding of the need for and potential benefit of 
conducting an implementation evaluation in your own 
setting. 

^Br€ak into small discussion groups Cthree to five per- 
sons); if possible, get together with persons who have a 
similar institutional role (i.e., all vocational directors 
group together, principals, department chairpersons, 
teachers, and so on). Then briefly discuss each of the 
following : • 

• Major decisions you've made over the last six months 
/ to a year and how you arrived at them. 

• -c " 

* How the decision-making grocessei^used rtight hape been 
more effective. (Could you have used more concrete 
data?) 
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Kinds of accountability demands you presently face. 



Types of 
you want 



decisions you need to make and/or questions 
to answer over the next six months to a year. 



Keep your dikcussion in mind as you proceed to the next 
activity in which you plan your own evaluation. 

The purpose of this activity is to give you an opportunity 
to talk to persons in the field who have been involved in 
evaluation issues. Record your findings and analyze them 
in terms of the issues raised below. 

Select a nearby high school or community college district, 
and interview several members of the vocational education 
staff to determine answers to the following questions: 

• How much of the district's vocational education budget 
is available and used for evaluation activities? 

• Does the district evaluate the effectiveness of its 
vocational instruction in terms of immediate student 
outcomes? If yes, when was it last done? How was it 
done (design, sample, measurement approaches, data 
anal/sis methods, results)? 

Critique your findings in terms of what you have learned 
in this module (i.e., assess the adequacy of the evalua- 
tion methods used; suggest ways in which it might have 
been better). \ 

NOTE: Make su^e. that* only one group of students inter- 
views each ^individual ♦ 
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Summary 

After the goals, objectives, and instructional strategies 
of a vocational education curriculum have been developed, the 
curriculum is ready to be implemented. This is also the time 
to begin determining whether the curriculum is effective in 
achieving its intended gpals and objectives — or outcomes. This 
process is known as evaluation : , the focus of this module. 
This module covers thrae main aspects of evaluation: • 

• The role of the curriculum specialist in evaluating 
vocational education curricula 

• The purposes and activities involved in evaluating 
curricula prior- to their implementation 

• The purposes and activities invofved in evaluating 
, curricula after they have been implemented 



The latter two aspects relate, in this module, to evaluating 
the short-term outcomes of vocational education curricula. 

Three approaches to vocational education evaluation are 
described: the accreditation approach, the accountability 
approach, and the comprehensive program evaluation approach. 
The curriculum specialist's, role requires a focused and modi- 
fied comprehensive program evaluation approach and involves 
collecting d'ata and making decisions in order to improve the 
curriculum and judge its worth. To accomplish these tasks, it 
is necessary to examine the planning and development of the 
curriculum, the way it is being used, and the outcomes it pro- 
duces. This examination, or assessment, should ideally take . 
place prior to and after any large-scale implementation of the 
curriculum. V* 

Pre-implementation evaluation acc&vities, or tryouts, are 
usually, carried out in'order to improve the curriculum before 
it Jt^used on a large scale. Implementation evaluation activi- 
ties, on the other hand, are conducted after the curriculum has 
been used more widely in order to assess the effectiveness of 
the curriculum in attaining its intended outcomes and to ■ 
facilitate decisions regarding its future. 

Prior to conducting any evaluation, it is important to 
review the intended curriculum outcomes, analyze^Jrtfe^logic 
li^i'Pg^he proposed activities to the outcomes, and, if neces- 
sary, divide the curriculum into components for separate analy- 
sis. Subsequently, the five steps listed below are performed 
in either the pre-implfementation or implementation phase of 
evaluation: 

t • * 
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^Planning the evaluation 

Developing process and outcome measures 
Selecting an evaluation design and sample(s) 
Conducting the. evaluation 
Processing and ^Jfcfcjtj-Og the data' 




^Two basic types of -measures^are likely to be used in voca- 
tional education evaluations: written tests and performance 
tests/ They "should be designed to adhere as closely as possible 
to five criteria of adSquicy: ' pbjectivity h reliability, vali- 
dity, efficiency, and non-teactivity . The persons tested should 
be representative of the intended audience, preferably selected 
randomly for the implementation phase. Resulting data may be 
processed and analyzed by* hand or machine (computer), depending 
on available resources and* evaluation needs. Once the data are 
analyz^i, they must be reported and utilizec}. This phase of the 
evaluation is covered in the module in thisVsesies devoted to 
conducting follow-up studies and communicating evaluation 
results* 



& 



V 
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Study Activity Responses 



GOAL 1 

Individual Study Activity Responses 

1. The purpose of the assigned reading was to provide an over 
view of the various types of vocational education evalua- 
tion and to stimulate you to consider the relationship of 
curriculum evaluation to program evaluation and materials 
evaluation* The^ scope of curriculum evaluation is nar- 
rower than that- of program evaluation, but it includes 
more than the evaluatjpn of the materials 1 quality. Cur- 
riculum evaluation's concerned with the planning and 

« development of .^curriculum, its implementation, and the 
short^ and long-term "outcomes it produces.* 

2. The purpose of the accreditation approach to evaluation is 
to ensure that educational programs and institutions meet 
minimum quality standards and achieve their goals. The 
purpose of the accountability approach to evaluation is to 

* provide evidence that educational funds are wiseLy spent 
for intended purposes. The purpose of the comprehensive 
'pr&^ram evaluation approach is- to improve educational pro- 
grams and to make decisions about their worth. 

3* A purriculum specialist should conduct an evaluation of a 
curriculum he or she has produced in order to identify 
ways to improve it and .to determine whether its outcomes ^ 
are of sufficient value to justify its continued imple- 
mentation* ' 

*■ 

3 a. 

2 -b. ' . . * 

JL c - • • . 

l d. \ 

3 



s 



_3_ f. 

JL 8: 
1 h. 
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Discussion Quest ion Responses 

— •% 

1. The discussion should include evaluation defined as: 
measurement; * 
congruence between performance and objectives; 
professional judgment; 
description; and 

provision T of information for decision-making, 

Xhe distinction between evaluation and evaluative research 
should be mentioned, as should the distinction between 
evaluation defined as the provision of information for 
judging decision alternatives vs., the determination of 
worth. To be consistent with evaluation concepts 
'preseTnted in this module, the definition of evaluation 
that the group evolves' should contain the following 
concepts. Evaluation is: 

• an ongoing process; 

• concerned with curriculum planning r , development , 
implementation', and short- and long-term outcomes; and 

r 

• conducted' for the purpose of improving the curriculum 
and judging it^ worth, 

j2. Responses will vary depending on the examples -the group 
members present. Hopefully, evaluations that include 
components representing all three evaluation approaches 
(accreditation, accountability , and comprehensive ^program 
evaluation) wiLl be offered, 

3.- , Federal" requirements have placed pressure on state and 
loc^l sjchobl systems to evaluate vocational education 
programs, but the lack of guidelines and trained personnel 
' has resulted in confusion. University personnel have 
'attempted to conceptualize the evaluation process into 
' models, but the application of theory to operating pro- 
grams has received less of thjeir attention. States are 
now instituting data collection" systems that may provide 
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information for evaluation and certainly will increase the 
reporting burden locally. On the whole, it is too early* 
to tell whether federal vocational education evaluation 
requirements' wj. II promote a process that, in the end, will 
produce benefits for vocational education students, 

4. A !, comprehens|ve program evaluation 1 ^ is possible in 

theory, but may require more resources (time, money, and 
talent) than are available. Furthermore,, the benefits of 
such an evaluation may not be sufficient to justify 
expending the necessary resources. Even if questions of 
worth are set a*side, problems of implementing, coordinat- 
ing, and making sense of the results of all aspects .of a 
comprehensive program evaluation remain. Given the cur- 
rent state-of-the-art in evaluation, we are not certain 
tlfat comprehensive program evaluation is possible. 



5. The major difference is in intent, not in procedures. 

However, it is unlikely that an evaluation would be con- 
ducted for solely one purpose. In reality, decisions 
aimed at improving a curriculum interact- with those aimed 
at judging its worth. Typically, evaluation activities 
conducted to inform those decisions are not clearly dis- 
tinguishable, either.- 



6. These three areas of vocational education curriculum 

evaluation should be conducted in sequence. A productive 
evaluation of curriculum outcomes must assume adequate 
curriculum implementation. Assessing curriculum planning 
and development should be cpnducte^l first to ensure that 
the curriculum is worth implementing. 



GOAL 2 



Individual. Study Activity Responses 
1. 
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a* 
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b. 






0> 


c . 






p 


d. 
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e. 
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f. 
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g- 
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2. See Wentling, T. L. Evaluating occupational education and 
training programs . Boston: Allyn and Bacofa, 1980. 

3. a. 3 

b. 5 

c. JL_ . 

d. 1 i 
•e. 5 * 

g- _2_ 

h. _4_ ; 

i. J_ ' 

j- _2_ ' ' . _ 

Discussion Question Responses 



1* a* Examples of everyday evaluations might include the 
following: 

• Color of tie or scarf to wear with shiYt or blouse 

• The" fit of one's clothing t 

• Degree of danger in driving on fre,£way« , 

The criteria for these everyday evaluations might 
include the following: 

• Hues and tones of clothing and their coordination 

• Conformance to body contours, tightness, or 
looseness 

• One's driving experience, degree of congestion of 
the freeways \j 

Purposes of everyday evaluations: 'Most evet*yday 
evaluations are conducted to improve one's appearance, 
increase comfort > imptovej the use of time, and,^ in 
some cases, to preserve one's safety. . 

b. Some everyday evaluations that a vocational teacher 
might carry out include the fo lowing: r 
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• The relative safety or cleanliness of the classroom 

• The effectiveness of a particular type of instruc- 
s tion in a given situation 

• j The type of material to use in a classroom 

The criteria for these evaluations might include the 
following : 

The amount of clutter around work arfeas , number of 
students, students 1 attitudes toward' safety 

% 

•* The number of ^students present , the degree of 

complexity of the concept or process being taught 

• The expense of materials, the type of materials 
used in the actual work environment, the danger 
involved in the use of certain materials 

Tftt^urposes of these evaluations : Most everyday^ 
e^alcmtions in vocational education are designed to 
improve the immediate instruction, to ensure safety, 
or to make the learning situation as realistic as 
possible. 



Response should indicate that pre-implementation evalua- 
tion is appropriate whenever • there are the desire and the 
resources to improve a curriculum prior to its large- 
scale implementation. Limited resources might constrain 
evaluation activities. Priorities must be selected. 

Assessing outcomes means determining whether intended out- 
comes were achieved, whereas assessing processes means 
determining whether the planned activities are being car-* 
ried out as intended. It is important to assess both in 
order to be able to attribute the attainment (or non- 
attainmeqt) of outcomes to proper (or improper) implement 
tation of the curriculum. 

Response should include ideas such as^the , following: 

After the exercises |are ready to be implemented, try them 
out on at least five (no more than ten) students,- using 
the same directions and conditions for all. Record the 
results of the exercises. Try to obtain some feedback 
from the student? themselves and other instructors. Tally 
the results. Use the results to determine whether the 
exercises are worthwhile; whether they need revisions; if 
so, what revisions should be made? 
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GOAL 3 

i 

Individual Study Activity Responses 

2. Response depends on individual's experiend/. 

3. The randomized] pretest-posttest control group design would 
probably acqorrahodate the situation. Three of the treat- 
ments would be instruction with each of the simulators, 
one treatment would be just as it always has been (without 
a simulator), and one group might receiyey a treatment con- 
sisting*^ a new program without a simulator. Selection 
of group members should be! random. 

4. Responses depend on 'individual's experiences. , 



Discussion Question Responses 

1. Discussion should focus on truth of the statement: e.g., 
trivial outcome data do not facilitate important, deci- ~ - 
sions; all outcomes should be->measurable , even difficult 
ones. 

2. Collecting data on how the curriculum was implemented 
helps determine whether it was implemented properly and 
whether outcomes can legitimately be related to the curri- 
culum. It helps future users implement it properly and 
provides a sound basis for predicting the effects of spe- 
cific activities on specific audiences and conditions. 

3. Discussion should focus on aspects of agreement to the 
statement. When no measures are used, ho information 
results. Some measures yield at least some information, - 
though it is important to cite the limitations of such 
data. Mention might be made that it is important to try 
to improve the adequacy of the measures. 

4. Practical problems might exist such as inadequate numbers 
*of available students, difficulty an pulling only certain 

students from existing groups, difficulty gaining partici- 
pation t>f random control groups, and so forth. 

5. Factors include resources (available money, personnel, 
time) and the complexity of the data and evaluation needs. 
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Self-Check 



1 



GOAL 1 1 

1. 'List and briefly describe three approaches to vocaTional 
education evaluation, ^ 

t 

2. Define' the curriculum specialist's role in vocational 
education evaluation. 



3. List the three essential areas of the evaluation of a 
vocational education curriculum.- 



GOAL 2 

1. List three analytical activities that should be done prior 
to conducting, an evaluation of vocational education cur- 
ricula. 

2. The main purpose of conducting preliminary tryouts of a 
vocational education curriculum is to . 

3. ^JList, the five steps involved in conducting preliminary 

tryouts of a vocational education curriculum. 

4. Identify the two basic types of tests likely to be used in 
evaluating 'vocational) education curricula. 

5. The evaluation design most appropriate for use in prelimi- 
nary tryouts is the ; 

6. The sample of persons involved in tryouts is likely to 
number 




c. 25 



d. 2 classes 





\ 



?♦ Data from preliminary tryouts are most likely to be pro- 
cessed by 

a* hand 

b. 'card sorter 

c. computer 

d. a combination of the three 
GOAL 3 

1. The main purpose of conducting an implementation* 
evaluation of a vocational education curriculum is 



to 



2. List the five steps involved in conducting an implementa- 
\^ tion evaluation of a vocational education curriculum. 

3. \dentify three types o*f data that should be sought from an 
iluation of a vocational education curriculum. 

Identify five characteristics (considerations of adequacy) 
of goo&*jneasurement techniques. 

5. The evaluation design most appropriate for use in an 

implementation evaluation is one in which • 



6. The sample used in an implementation evaluation should 
be of the defined audience and should 

be selected. 

7. The number of persons in each group tested should be no 
lower than 

a. 10 

b. -25 

c. 50^__^-^ 

d. 100 

8. List three ways in which descriptive "data may be presented, 
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Self-Check Responses 



The answers that follow will give you an idea of the types 
of response expected. Use them ?s a study tool if - you wish. 



GOAL 1 



1« Accreditation approach : Self study and peer review con- 
ducted to ensure that educational programs and institu- 
tions meet minimu^f quality standards and achieve goals . 

Accountability approach : Aggregate data collected and 
reported to show that educational funds are wisely spent 
for intended purpose^ 

Comprehensive program evaluation approach : Examination of 
program planning, operation, aj p^ outcome s using input from 
many sources conducted to improve* the program or judge its 
wbrth, 

2. to collect data and m^ke decisions to improve his or her 
product (the curriculum) and judge its worth 

3. An assessment of Curriculum planning and development 
An assessment of curriculum implementation 

An assessment of the outcomes of a curriculum 



GOAL 2 

j # 

• / 

1. • Review intended curriculum \outcomes. 

• Review logic linking proposed activities to desired 
outcomes. 

• Divide curriculum into components, if necessary. 

2. ...help improve the curriculum prior to and during its 
early implementation; to improve the ultimate product. 

3. • Plan the tryouts. 

♦ • Develop process and outcome Measures. 

• Select a sample and an evaluation design. 

• Conduct the tryouts. 

• Process, analyze, and use data to revise /curriculum. 

4. • Written tests 

• Performance tests 

5» ♦.. one-group pretest-posttest design 
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6. a 



7. 



a 



GOAL 3 
2. 



3. 



6. 



..assess the effectiveness of the curriculum in attaining 
ntende^ outcomes and to facilitate decisions regarding 
he future of the curriculum. 

Plan the evaluation. 
Develop pleasures. 

Select an evaluation design and sample. 
Conduct the evaluation (collect the data). 
Process and analyze the data. 

'Outcome data 
Process data 

Data on unintended side effects (outcomes) 

-Objectivity 
Reliability 
Validity 
Efficiency 
Non-reactivity 



Outcome measures (tests) are administered to a control and 
an experimental group both before (pretest) and after 
(posttest) ' the curriculum is implemented. 

. . .representative 
. . . randomly 



7. 
8. 



Tables L 
Statistical summaries 
Graphs 
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